Learning to reach and reaching to learn: a unified approach to path planning and reactive control through reinforcement learning

The next generation of intelligent robots will need to be able to plan reaches. Not just ballistic point to point reaches, but reaches around things such as the edge of a table, a nearby human, or any other known object in the robot’s workspace. Planning reaches may seem easy to us humans, because we do it so intuitively, but it has proven to be a challenging problem, which continues to limit the versatility of what robots can do today. In this document, I propose a novel intrinsically motivated RL system that draws on both Path/Motion Planning and Reactive Control. Through Reinforcement Learning, it tightly integrates these two previously disparate approaches to robotics. The RL system is evaluated on a task, which is as yet unsolved by roboticists in practice. That is to put the palm of the iCub humanoid robot on arbitrary target objects in its workspace, start- ing from arbitrary initial configurations. Such motions can be generated by planning, or searching the configuration space, but this typically results in some kind of trajectory, which must then be tracked by a separate controller, and such an approach offers a brit- tle runtime solution because it is inflexible. Purely reactive systems are robust to many problems that render a planned trajectory infeasible, but lacking the capacity to search, they tend to get stuck behind constraints, and therefore do not replace motion planners. The planner/controller proposed here is novel in that it deliberately plans reaches without the need to track trajectories. Instead, reaches are composed of sequences of reactive motion primitives, implemented by my Modular Behavioral Environment (MoBeE), which provides (fictitious) force control with reactive collision avoidance by way of a realtime kinematic/geometric model of the robot and its workspace. Thus, to the best of my knowledge, mine is the first reach planning approach to simultaneously offer the best of both the Path/Motion Planning and Reactive Control approaches. By controlling the real, physical robot directly, and feeling the influence of the con- straints imposed by MoBeE, the proposed system learns a stochastic model of the iCub’s configuration space. Then, the model is exploited as a multiple query path planner to find sensible pre-reach poses, from which to initiate reaching actions. Experiments show that the system can autonomously find practical reaches to target objects in workspace and offers excellent robustness to changes in the workspace configuration as well as noise in the robot’s sensory-motor apparatus

From Hubbard bands to spin-polaron excitations in the doped Mott material Nax_xCoO2_2

We investigate the excitation spectrum of strongly correlated sodium cobaltate within a realistic many-body description beyond dynamical mean-field theory (DMFT). At lower doping around $x$=0.3, rather close to Mott-critical half-filling, the single-particle spectral function of Na$_x$CoO$_2$ displays an upper Hubbard band which is captured within DMFT. Momentum-dependent self-energy effects beyond DMFT become dominant at higher doping. Around a doping level of $x\sim 0.67$, the incoherent excitations give way to finite-energy spin-polaron excitations in close agreement with optics experiments. These excitations are a direct consequence of the formation of bound states between quasiparticles and paramagnons in the proximity to in-plane ferromagnetic ordering.Comment: 5 pages, 3 figures; supplementary materia

Взаємодія напівпровідників типу АІІІВV з розчинами Н2О2 - НВr

To plan complex motions of robots with many degrees of freedom, our novel, very flexible framework builds task-relevant roadmaps (TRMs), using a new sampling-based optimizer called Natural Gradient Inverse Kinematics (NGIK) based on natural evolution strategies (NES). To build TRMs, NGIK iteratively optimizes postures covering task-spaces expressed by arbitrary task-functions, subject to constraints expressed by arbitrary cost-functions, transparently dealing with both hard and soft constraints. TRMs are grown to maximally cover the task-space while minimizing costs. Unlike Jacobian-based methods, our algorithm does not rely on calculation of gradients, making application of the algorithm much simpler. We show how NGIK outperforms recent related sampling algorithms. A <font color="blue"><a href="http://youtu.be/N6x2e1Zf_yg">video demo</a></font> successfully applies TRMs to an iCub humanoid robot with 41 DOF in its upper body, arms, hands, head, and eyes. To our knowledge, no similar methods exhibit such a degree of flexibility in defining movements

ALife in humanoids: Developing a framework to employ artificial life techniques for high-level perception and cognition tasks on humanoid robots

We describe our recent research and advances in building a framework enabling artifical life (ALife) systems on real robotic hardware. Our framework allows our iCub humanoid to build better visual perception, improve its motion capabilities and even provide a sense of proprioception. This paper presents how we can use various techniques, such as, e.g., genetic programming, to build subsystems for these specific areas. Our framework runs in parallel with the hardware system and is updated with new information from the real robot. We plan to use this framework in the future for developing higher cognitive tasks, such as, scene understanding, prediction of action outcomes, and reasoning on our robot

Towards Spatial Perception: Learning to Locate Objects From Vision

Our humanoid robot learns to provide position estimates of objects placed on a table, even while the robot is moving its torso, head and eyes (cm range accuracy). These estimates are provided by trained artificial neural networks (ANN) and a genetic programming (GP) method, based solely on the inputs from the two cameras and the joint encoder positions. No prior camera calibration and kinematic model is used. We find that ANN and GP are both able to localise objects robustly regardless of the robot's pose and without an explicit kinematic model or camera calibration. These approaches yield an accuracy comparable to current techniques used on the iCub

Rapid Fluvio-Thermal Erosion of a Yedoma Permafrost Cliff in the Lena River Delta

The degradation of ice-rich permafrost deposits has the potential to release large amounts of previously freeze-locked carbon (C) and nitrogen (N) with local implications, such as affecting riverine and near-shore ecosystems, but also global impacts such as the release of greenhouse gases into the atmosphere. Here, we study the rapid erosion of the up to 27.7 m high and 1,660 m long Sobo-Sise yedoma cliff in the Lena River Delta using a remote sensing-based time-series analysis covering 53 years and calculate the mean annual sediment as well as C and N release into the Lena River. We find that the Sobo-Sise yedoma cliff, which exposes ice-rich late Pleistocene to Holocene deposits, had a mean long-term (1965–2018) erosion rate of 9.1 m yr–1 with locally and temporally varying rates of up to 22.3 m yr–1. These rates are among the highest measured erosion rates for permafrost coastal and river shoreline stretches. The fluvio-thermal erosion led to the release of substantial amounts of C (soil organic carbon and dissolved organic carbon) and N to the river system. On average, currently at least 5.2 × 106 kg organic C and 0.4 × 106 kg N were eroded annually (2015–2018) into the Lena River. The observed sediment and organic matter erosion was persistent over the observation period also due to the specific configuration of river flow direction and cliff shore orientation. Our observations highlight the importance to further study rapid fluvio-thermal erosion processes in the permafrost region, also because our study shows increasing erosion rates at Sobo-Sise Cliff in the most recent investigated time periods. The organic C and N transport from land to river and eventually to the Arctic Ocean from this and similar settings may have severe implications on the biogeochemistry and ecology of the near-shore zone of the Laptev Sea as well as for turnover and rapid release of old C and N to the atmosphere

Modern rates of thermal denudation and thermal abrasion on western Kolguev Island

Destruction mechanisms and dynamics of the Arctic coast, also in the western sector of the Russian Arctic, are studied in detail, including the use of remote sensing data. However, data on thermal abrasion and thermo denudation of Kolguev island is quite limited. Some estimates were presented in article of M.A.Velikotsky (1998). Estimation of thermos denudation rates near the Sauchiha river mouth for the period 1948-2002 years was done by the authors earlier (Kizyakov& Perednya, 2003). To obtain data about the modern (after 2002) shoreline retreat rates and growth of thermal cirque a high resolution remote sensing data were involved in our research. Part of the western coast of the Kolguev island was inspected in field work conducted on 2002 by ECI SB RAS, together with VNIIOkeangeologia. The object of research was the part of coast, including a group of three coastal thermal cirques, located 3.5 km south of the Sauchiha river mouth. In 2012, within the framework of the project ‘Geoportal of MSU’ operational satellite imaging was done on Kolguev island by satellite FORMOSAT-2. High resolution satellite imagery provides ample opportunities for visual interpretation of coastal landforms. Aerial photographs (1948 and 1968), surveying materials (2002), high-resolution satellite images (2009 and 2012) became basis to study the dynamics of the coast and thermal cirques in the key area. For key area were calculated: retreat rates of the edge of the coastal terraces and thermal cirques for the periods 1948-1968, 1968-2002, 2002-2009, 2009-2012; retreat rates of the foot of the coastal terrace for the periods 2002-2009, 2009-2012; volume of the material enters the coastal zone by the thermal abrasion for one linear km of a coast (Kizyakov et al., 2013). Average long-term rates of retreat of the coastal terrace during 1948-2012 varied from 0.7 to 2.4 m/year; 2002-2012 varied from 1.7 to 2.4 m/year. Identified rates are distinctive for the part of coast from the mouth of Krivaya river to the curve of coastline near the mouth of the Gusinaya river - a length is 60.5 km. These rates are in 1.1-1.5 times lower than average rates of retreat of thermal cirque edges which are connected with melting of massive ice deposits. Averaged growth rates of the thermal cirques in 1948-2002 was 2.4 m/year; in 2002-2012 was - 2.6 m/year. The maximum growth rate on some sections in 2009-2012 were 14.5-15.1 m/year. These rates are the largest for the previously recorded in the Western sector of the Russian Arctic. The cause of the abnormally high rates is an increase the annual amount of positive air temperatures, which in 2011-2012 was 1.4-1.5 times higher than the long-term average. The determined rates of the development of thermal cirque can be extended to the north from the key area (near the Sauchiha river mouth) to the Gusinaya river mouth with total length of 32.3 km. The next plans on studying the coastal dynamics on Kolguev Island - using additional satellite images for the purposes of: detailization of interannual dynamics through the analysis of more short time span series of satellite images, definition of variations of the coastal destruction rates on the Western and Northern coasts. References: Velikotsky M.A. Characteristics of modern coastal dynamics of the Kolguev Island // Dynamics of the Russian Arctic coasts, Moscow, MSU – 1989 – P.93- 101 (In Russian) Kizyakov A.I., Perednya D.D. Destruction of coasts on the Yugorsky Peninsula and on Kolguev Island (Russia) // Permafrost: Abstr. of the 8th Intern. Conf. (Zurich, Switzerland, 21–25 July 2003). Zurich, Switzerland – 2003 – P. 79–80. Kizyakov A.I., Zimin M.V., Leibman M.O., Pravikova N.V. Monitoring the rate of thermal denudation and thermal abrasion on the western coast of Kolguev Island using high resolution satellite images // Earth Cryosphere (Kriosfera Zemli). – 2013, XVII, No. 4 – P. 15-25 (In Russian)